Theory and Credibility: Integrating Theoretical and Empirical Social Science

Theory and Credibility: Integrating Theoretical and Empirical Social Science

  • Downloads:1662
  • Type:Epub+TxT+PDF+Mobi
  • Create Date:2021-10-12 06:52:09
  • Update Date:2025-09-07
  • Status:finish
  • Author:Scott Ashworth
  • ISBN:0691213828
  • Environment:PC/Android/iPhone/iPad/Kindle

Summary

A clear and comprehensive framework for bridging the widening gap between theorists and empiricists in social science



The credibility revolution, with its emphasis on empirical methods for causal inference, has led to concerns among scholars that the canonical questions about politics and society are being neglected because they are no longer deemed answerable。 Theory and Credibility stakes out an opposing view--presenting a new vision of how, working together, the credibility revolution and formal theory can advance social scientific inquiry。

This authoritative book covers the conceptual foundations and practicalities of both model building and research design, providing a new framework to link theory and empirics。 Drawing on diverse examples from political science, it presents a typology of the rich set of interactions that are possible between theory and empirics。 This typology opens up new ways for scholars to make progress on substantive questions, and enables researchers from disparate traditions to gain a deeper appreciation for each other's work and why it matters。

Theory and Credibility shows theorists how to create models that are genuinely useful to empirical inquiry, and helps empiricists better understand how to structure their research in ways that speak to theoretically meaningful questions。

Download

Reviews

David Childers

This is a book that needed to be written, about reconciling theory and empirics, but, more specifically, the kind of theory and empirics that get done in contemporary quantitative political science and applied microeconomics。 "Theory" here means the kind of simple rational choice and game theoretic models in the style of Gary Becker consisting of a small set of agents, some simple descriptions of their objective functions and choice sets, and maybe an equilibrium structure; "credibility" means s This is a book that needed to be written, about reconciling theory and empirics, but, more specifically, the kind of theory and empirics that get done in contemporary quantitative political science and applied microeconomics。 "Theory" here means the kind of simple rational choice and game theoretic models in the style of Gary Becker consisting of a small set of agents, some simple descriptions of their objective functions and choice sets, and maybe an equilibrium structure; "credibility" means specifically the causal econometric tool set of Angrist and Pishke's "Mostly Harmless Econometrics" underlying their "credibility revolution" in favor of experimental and quasi-experimental methods for inferring effects of a treatment。 Though brief reviews are provided and the style is informal and not strongly technical, the target reader should have an easy familiarity with each, with the ability to follow a game tree to find a subgame perfect equilibrium and also reason about the exclusion restriction in an instrumental variables estimator。 This reflects the book's origins in a PhD course but also the background of a huge swathe of working economists and political scientists, who have been put through this same training and regularly read and write papers using one or both of these traditions。 The question this book tries to answer, both through a more abstract framework of interplay between model and empirics and through extensive examples drawn from recent quantitative political science research, is how to reconcile what seem to be fairly strong tensions between these two traditions and apply them in complementary ways to actually learn about social scientific phenomena such as elections, protests, and civil conflicts。That there is a tension between the way social scientists do theory and empirics will be obvious to anyone who's ever sat through a seminar or, worse, hiring committee argument about respective value。 To the more dogmatic "credibility revolution" style empiricists, the "toy" models written by theorists often fail to account for the variety of heterogeneous forces that may be influencing some behavior, spelling out some simple "just so" story of this leading to that to come to some pre-baked and maybe ideologically convenient conclusion, while a fuller account could just as easily go the other way。 A formative experience in my education was sitting in the back of the seminar room while ur-experimentalist Abhijit Banerjee, visiting my institution, would patiently and methodically, after some hapless grad student or even more hapless tenured professor had laid out their model and conclusions, draw on his background as a theorist and lay out an alternative model showing that the theorist had neglected some obvious pathway or relied on some functional form assumption implicitly implying some empirically absurd elasticity, and have the audience and sometimes even the presenter convinced that the opposite conclusion was true, or at least sufficiently plausible that the issue remained unresolved。 In contrast, a theorist might now look out at huge swathes of work that has run experiments or searched for natural experiments for their own sake and find that little to nothing has been learned about the questions they got into social science to answer, with precise estimates of causal effects of interventions of limited inherent interest, with no meaningful interpretation in terms of broader paradigms or any pretense of applicability beyond the narrow context in which it is applied。 So, if our models are unreliable guides to outcomes and our empirics give us precise answers to the wrong questions, what should we be doing? Here is where one would expect our "structural" colleagues to jump in with their answer, which is to to meet in the middle, by extending the scope and realism of our theories until they encompass the competing and diverse forces that determine a particular outcome and tailor our empirical methods to bring our enriched model directly to estimate the model-implied quantities of interest while accounting for heterogeneity, unknown functional forms, and the presence of unobserved influences。 This is not at all where Ashworth, Berry, and Bueno de Mesquita go。^1 Instead, in best Chicago school form, they provide a rousing defense of the status quo, at least as applied with care and intelligence in its ideal form。 Simple theoretical models provide a way to succinctly express a mechanism, outlining one or more major forces and the way they interact to result in a particular set of outcomes, and as intuition pumps to think through the implications of such a mechanism, which may not be obvious without formalizing how the parts come together。 However, they do not and are not intended to provide a full or empirically complete accounting of the empirical behavior of any particular variable or set of variables; instead, they should be linked to empirical implications by being treated as descriptions of "all else equal" phenomena, that hold fixed the other elements of reality in order to make clear one particular implication。 In the examples, this is often some direct or indirect effect, made clear by following the logic that if an intentional agent cares about one thing as a primitive, it may induce choices regarding other things that have consequence for the former, or selection effects, whereby attributes with different consequences induce differing choices by agents heterogeneous with respect to those attributes。 Often left out of the formal model, however, are ancillary sources of heterogeneity or alternative mechanisms which may interact minimally with the proposed one。 It is also common to use highly simplified forms, such as linear or CES utility or response functions, uniform or Gaussian or discrete distributions, and so on, which make it easy to solve and analyze the implications of the model。 Aside from, as far as I can tell, a single exception, for which they apologize, no model they introduce takes more than about a page to describe and solve。 Although the term is not used in the text, one way of thinking about the role of such a model is as providing a subgraph of the larger DAG describing the full structure of causal relationships, with a model describing some links and possibly a few of their features like the sign, but making no pretense to describe the full structure or exactly quantify particular components。The point of such models is that one can derive from them salient and distinguishing empirical predictions that would describe the action of the mechanism all else equal。 This is often a causal effect, such as, holding everything else fixed, X should increase Y, but could be a correlation if that correlation is sufficiently distinguishing of a certain feature。 The role of empirical methods, and particularly credibility revolution style methods, is that in order to provide a test of such an implication, one does need to actually hold all else equal, by isolating variation from the source as described in the model。 One can also use these methods to reinforce or contradict a particular mechanism by examining additional downstream effects, or components of the mechanism, which should be active if the mechanism is indeed operative, and, given multiple mechanisms, possibly embodied in multiple models distinguish between and quantify their relative contributions。 The models provide guidance on which effects are important to test for, and can also provide intuition to understand whether the sources of variation used in the empirical model satisfy the assumptions needed to provide reasonable estimates。 Without credible empirical estimates of effects, one would have only weak and unpersuasive evidence in favor of particular features, but without a model, one would not be able to to explain the role or sources of those effects。The first half of the book lays out this argument abstractly; the second half makes it concrete by providing examples from political science research, on topics such as party control and voting, women's participation in politics, the benefits of incumbency, the economic motivations for civil conflict, and the effectiveness of protests。 For each they provide one or a few simple models, often with shared features across contexts, as in the model of candidate selection used in modified form as a component in models for the first three examples, and derive basic implications which can be evaluated in an "all-else equal" fashion using causal methods, then go back and assess how much the results told us about the forces in the models and what kind of additional empirical results or modifications to the theory might be needed to bring this forward。 In some cases, the model tells us about the adequacy of a particular method, by highlighting confounding factors, violations of the exclusion restriction, and so on。 In other cases it highlights what to focus on; for example, in the economic shocks and conflict example, it is noted that early causal work using exogenous commodity price variation found negligible average effects on conflict, but a look at the theory of why this might be suggested two opposing causal mechanisms pointing in opposite directions; first that higher prices may raise the benefits to gaining economic control through rebellion, but second that higher prices may raise the opportunity costs of conflict for producers。 This led Dube and Vargas to attempt to isolate these mechanisms, in their study using a triple diff design comparing oil vs coffee price fluctuations for Colombia, as commodities that might differentially impact the former versus the latter mechanism; the theory also suggested auxiliary tests, such as measuring impact on wages, which could help confirm or disconfirm the claimed mechanisms。Overall, I found the book highly worthwhile as an exercise in thinking through what the role is for a model section in a paper, and what kinds of causal effects are worth investigating。 The theoretical framework they provide is somewhat loose, illustrated mostly with example rather than concrete definitions, and while seeing so many tiny but genuinely useful models and accompanying well-motivated empirical investigations provides good training for any social scientist wanting to continue on in this tradition, they are very much describing an art rather than a science。 In some cases, it felt as if the models, while useful for thinking through some issues, were merely providing illustrations of concepts fully subsumed by the causal inference framework, but with additional parametric assumptions that reduced generality。 Formal mediation analysis is mentioned, but described as requiring strong assumptions (it is known, for example, that even with experimental assignment of treatment that mediation effects may not be identified due to confounding of the mediator), and yet many of the practical discussions of disentangling or quantifying mechanisms involved exactly identifying quantities from the mediation literature like a natural or controlled direct effect, in the context of a model without bringing these assumptions in。 By bringing this back to the causal framework and explicitly laying out the mediation assumptions, one might better be able to see features that are being left out of the model, and might be important to account for empirically。 In the example of protest efficacy, the model of informational effects of protest could be described as implying a violation of the exclusion restriction for weather as an instrumental variable due to its impact on perceptions of the cost of protest。 In the discussion of Lee (2008)'s regression discontinuity estimates of the effect of incumbency, the argument that these estimates recover something other than what they call the "insulation effect" of incumbency was precisely a claim that the local average treatment effect identified by the discontinuity would differ from the population average treatment effect because candidate quality impacts both selection into close elections and moderates the effect of incumbency。 None of these is a failing of the model based approach or an illustration that it can be done away with, as none of these claims, which can be expressed fully in causal language, would be quite so clear a source of problems to investigate had they not gone through simple models that implied the results, but they do perhaps illustrate additional value added in a fully "bilingual" approach, which translates back and forth between model and estimator assumptions。 Overall, tension must remain between modeling and estimation approaches if one strives to keep them separate, as opposed to formulating a theory which makes empirical predictions expressed directly in terms of probability statements about observables, whether expressed in terms of a full generative model or partially as quantitative or even qualitative semiparametric implications。 As a result, the descriptions in this book involved some level of vagueness, with illustration primarily by example。 As a practical guide to how to live with this vagueness and make the best of it, building maximally useful models with a minimum of complexity and coming up with research designs aligned with the purpose of the model, I appreciate this book and will recommend it to my students。 Outside the hands of researchers with the level of artisanship of the authors, I worry that success may not be so easy to replicate, but this look at the thought process that goes into doing this style of research carefully should give us a better shot。 1: This is arguably for good reason, as the structural approach is difficult, time consuming, and easy to do badly, with the possibility of inheriting the downsides of both modern theory and empirics, by heaping on "bells and whistles" without addressing the substantive counteracting forces, and relying on estimates which are robust in all the ways except the ones that matter for your particular application。 Of course poorly done implementations don't speak to the value of properly done ones, and in any case this topic is not really addressed in this book, perhaps as these approaches currently appear to have limited purchase in quantitative political science。 。。。more